Add kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5 infer code by ZTWHHH · Pull Request #147 · mm-evaluation/simple-mmeval

ZTWHHH · 2026-06-10T15:25:01Z

Description

Add 6 new multimodal model series to the framework. All inference files default to max_new_tokens=2048. Companion test-result PR: #148.

Series	HF model	Tested variants	Modalities
`kimi_vl`	moonshotai/Kimi-VL-A3B-{Instruct,Thinking}	Both	text, multi-image-interleave
`minicpm_v_4d5`	openbmb/MiniCPM-V-4_5	All	text, multi-image-interleave
`glm_4d1v`	THUDM/GLM-4.1V-9B-{Thinking,Base}	Both	text, multi-image-interleave
`ovis2`	AIDC-AI/Ovis2-{1B,2B,4B,8B}	All registered	text, multi-image-interleave
`aria`	rhymes-ai/Aria	All	text, multi-image-interleave
`qwen3d5`	Qwen/Qwen3.5-{0.8B,2B,4B,9B}	All registered	text, multi-image-video-interleave

Qwen3.5 is the official unified vision-language family (Qwen3_5ForConditionalGeneration + Qwen3VLProcessor); see https://qwen.ai/blog?id=qwen3.5 and https://github.com/QwenLM/Qwen3.5.

This PR contains infer/model-code changes only.

Files:

mmeval/infer/{kimi_vl,minicpm_v_4d5,glm_4d1v,ovis2,aria,qwen3d5}.py
env_files/{kimi_vl,minicpm_v_4d5,glm_4d1v,ovis2,aria,qwen3d5}_requirements.txt
mmeval/registry.py

Type of Change

Testing

Tested locally
Added example script (if new model/dataset)

Test command used:

PYTHONPATH=. ENV_DIR=/raid/ztw/envs python mmeval/run.py \
  --model_name_or_path <HF model id> \
  --dataset local@json \
  --infile tests/samples/<modality>.json \
  --img_dir tests/media/448 \
  --out_dir <work_dir> \
  --gpu_per_parallel 1 --parallel_per_task 1 \
  --max_new_tokens 2048

Checklist

Updated mmeval/registery.py (if new model)
Updated README (if new model/dataset)
Code follows existing style

- mmeval/infer/{kimi_vl,minicpm_v_4d5,glm_4d1v,ovis2,aria}.py - env_files/{kimi_vl,minicpm_v_4d5,glm_4d1v,ovis2,aria}_requirements.txt - mmeval/registry.py: series_mapping + series_infer_env_mapping entries Models: - kimi_vl: moonshotai/Kimi-VL-A3B-Instruct, Kimi-VL-A3B-Thinking - minicpm_v_4d5: openbmb/MiniCPM-V-4_5 - glm_4d1v: THUDM/GLM-4.1V-9B-Thinking, GLM-4.1V-9B-Base - ovis2: AIDC-AI/Ovis2-{1B,2B,4B,8B,16B,34B} - aria: rhymes-ai/Aria Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Adds the official Qwen3.5 multimodal series to the same branch alongside the previous 5 series. Qwen3.5 is a unified vision-language model (no separate -VL suffix); architecture: Qwen3_5ForConditionalGeneration with Qwen3VLProcessor. Models: Qwen/Qwen3.5-{0.8B,2B,4B,9B,27B,35B-A3B,122B-A10B,397B-A17B} and Base / FP8 / GPTQ-Int4 variants. HF: https://huggingface.co/Qwen/Qwen3.5-9B GH: https://github.com/QwenLM/Qwen3.5 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

- ovis2: registered variants narrowed to Ovis2-{1B,2B,4B,8B} (all tested). Larger Ovis2-{16B,34B} variants are de-registered to keep the registry to tested-only variants. - qwen3d5: registered variants narrowed to Qwen3.5-{0.8B,2B,4B,9B}. Larger variants (27B / 35B-A3B / 122B-A10B / 397B-A17B and their Base/FP8/GPTQ flavors) are de-registered for the same reason; they can be reintroduced in a follow-up once tested. - env_files/glm_4d1v_requirements.txt: add av (PyAV) — the Glm4v processor's video path requires it via torchvision.io.

The previous parse_input concatenated all text fragments and appended them at the end of the content list, breaking interleave order for prompts where images appear between text chunks (e.g. 'Image A <image> and Image B <image>'). Append text chunks in position so the multi-image interleave path matches the official model.chat() expectations.

Previous defaults (512/1024/8192) replaced with a uniform 2048 for the six newly added series. 2048 is the project-wide normal upper bound; no model needs a lower limit.

Copilot

Pull request overview

Adds six new multimodal model series integrations to simple-mmeval, wiring each series into the registry and providing per-series inference runners plus environment requirement files (defaulting to max_new_tokens=2048).

Changes:

Register 6 new model series (kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5) in mmeval/registry.py and map them to env + infer entrypoints.
Add inference implementations for the 6 series under mmeval/infer/.
Add per-series environment requirement files under env_files/.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
mmeval/registry.py	Adds series name→model-name registry entries and env/infer mappings for the 6 new series.
mmeval/infer/kimi_vl.py	Adds inference runner for Kimi-VL-A3B variants (text + multi-image interleave).
mmeval/infer/minicpm_v_4d5.py	Adds inference runner for MiniCPM-V-4_5 (text + multi-image interleave).
mmeval/infer/glm_4d1v.py	Adds inference runner for GLM-4.1V (text + multi-image/video interleave).
mmeval/infer/ovis2.py	Adds inference runner for Ovis2 (text + multi-image interleave).
mmeval/infer/aria.py	Adds inference runner for Aria (text + multi-image interleave).
mmeval/infer/qwen3d5.py	Adds inference runner for Qwen3.5 unified VL family (text + multi-image/video interleave).
env_files/kimi_vl_requirements.txt	Defines per-series Python deps for `kimi_vl`.
env_files/minicpm_v_4d5_requirements.txt	Defines per-series Python deps for `minicpm_v_4d5`.
env_files/glm_4d1v_requirements.txt	Defines per-series Python deps for `glm_4d1v`.
env_files/ovis2_requirements.txt	Defines per-series Python deps for `ovis2`.
env_files/aria_requirements.txt	Defines per-series Python deps for `aria`.
env_files/qwen3d5_requirements.txt	Defines per-series Python deps for `qwen3d5`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ZTWHHH · 2026-06-17T03:29:49Z

+torch==2.6.0
+torchvision==0.21.0
+transformers @ git+https://github.com/huggingface/transformers.git@main
+accelerate


Thanks for the suggestion. I considered the reproducibility benefit, but reverted the fixed transformers commit pin to keep the integration closer to the official Qwen3.5 dependency setting (the model card explicitly asks for the latest transformers from main). Since there is no confirmed functional issue requiring a pinned internal commit, I prefer to follow the official setup here.

ZTWHHH · 2026-06-17T03:30:18Z

+        text = self.processor.apply_chat_template(
+            user_message, add_generation_prompt=True, return_tensors="pt",
+        )
+        proc_kwargs = {"text": text, "return_tensors": "pt", "padding": True, "truncation": True}
+        if images:
+            proc_kwargs["images"] = images
+        inputs = self.processor(**proc_kwargs).to(self.model.device)


Thanks for the review. I checked this against the official Kimi-VL README, which uses the same processor.apply_chat_template(..., return_tensors="pt") → processor(images=image, text=text, ...) pattern; the Kimi-VL processor accepts pre-tokenized ids here. The tests pass end-to-end, so I kept the official pattern to preserve upstream fidelity.

ZTWHHH and others added 2 commits June 10, 2026 15:23

ZTWHHH changed the title ~~Add 5 new model series: kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria~~ Add kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5 infer code Jun 10, 2026

ZTWHHH mentioned this pull request Jun 11, 2026

Add kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5 test scripts and results #148

Merged

10 tasks

ZTWHHH added 2 commits June 11, 2026 08:10

Raise default max_new_tokens to 2048 for the 6 new inference files

f115d88

Previous defaults (512/1024/8192) replaced with a uniform 2048 for the six newly added series. 2048 is the project-wide normal upper bound; no model needs a lower limit.

ZTWHHH requested a review from Irisicy4 June 14, 2026 15:12

Irisicy4 requested a review from Copilot June 14, 2026 15:15

Copilot started reviewing on behalf of Irisicy4 June 14, 2026 15:15 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

ZTWHHH force-pushed the add-five-new-model-series branch from 215af03 to 2911c6b Compare June 17, 2026 02:55

ZTWHHH force-pushed the add-five-new-model-series branch from 2911c6b to 28f7071 Compare June 17, 2026 03:19

ZTWHHH merged commit f2d672a into mm-evaluation:main Jun 17, 2026
1 of 7 checks passed

Navigation Menu

Uh oh!

Add kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5 infer code - #147

Add kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5 infer code#147
ZTWHHH merged 6 commits into
mm-evaluation:mainfrom
ZTWHHH:add-five-new-model-series

ZTWHHH commented Jun 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

ZTWHHH Jun 17, 2026

Uh oh!

ZTWHHH Jun 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ZTWHHH commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of Change

Testing

Test command used:

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

ZTWHHH Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

ZTWHHH Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ZTWHHH commented Jun 10, 2026 •

edited

Loading